Stein— Sahi complementary series 
and their degenerations 

Yuri A. Neretii^ 

c7^ ' The paper is an introduction to the Stein-Sahi complementary series, and the 

^rj , unipotent representations. We also discuss some open problems related to these ob- 

jects. For the sake of simplicity, we consider only the groups U(n,n). 
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1 Introduction 



This papeiQ is an attempt to present an introduction to the Stein-Sahi comple- 
mentary series available for non-experts and beginners. 

p-H ' 1.1. History of the subject. Theory of infinite dimensional representa- 

p^ . tions of semi-simple groups was initiated in pioneer works of L M. Gelfand and 

M. A. Naimark (1946-1950), V. Bargmann (1947), and K. O. Friedrichs [12] 
(1951-1953). The book [H] by I. M. Gelfand and M. A. Naimark (1950) con- 
tains a well-developed theory for complex classical groups GL(n, C), SO(n,C), 
Sp(2n,C) (the parabolic induction, complementary series, spherical functions, 
characters, Plancherel theorems). However, this classical boolO contained var- 

^Sl ■ ious statements and asseverations that were not actually proved. In the mod- 

^ ' ern terminology, some of chapters were 'mathematical physics'. The most of 

^D , these statements were really proved by 1958-1962 in works of different authors 

^^ ' (Harish-Chandra, F. A. Berezin, etc.). 

In particular, I. M. Gelfand and M. A. Naimark (1950) claimed that they 
classified all unitary representations of GL(n,C), SO(n,C), Sp(2ri, C). E. Stein 
f46 compared Gelfand-Naimark constructions for groups SL(4, C) ~ S0(6, C) 

QQ ' and observed that they are not equivalent. In 1967 E. Stein constructed 'new' 

f^ , unitary representations of SL(2n, C). 

D. Vogan [48J in 1986 obtained the classification of unitary representations of 

groups GL(2n) over real numbers R and quaternions H. In particular, this work 

contains extension of Stein's construction to these groups. In 1990s, the Stein- 

^ . type representations were a topic of interest of S. Sahi see [4^ , |41] , [42] , S. Sahi- 

E. Stein 44J, A. Dvorsky-S. Sahi [8]-[9j. In particular, Sahi extended the con- 
struction to other series of classical groups, precisely to the groups S0(2n, 2n), 
U(n,n), Sp(n,n), Sp(2n,R), S0*(4n), Sp(4n,C), and S0(2n,C). 

1.2. Stein-Sahi representations for \J{n,n). Denote by U(n) the group 
of unitary n x n-matrices. Consider the pseudo-unitary group \J{n,n). We 

realize it as the group of {n + n) x {n + ri)-niatrices g = ( ,] satisfying the 
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Lemma 1.1 The formula 

z^ z^3] .-{^a + zc)-^{h + zd) (1.1) 

determines an action of the group U(n,n) on the space U(n). 

The unitary group is equipped by the Haar measure dfi{z), hence we can 
determine the Jacobian of a transformation ()1.1|) by 

dii(zj 

Lemma 1.2 The Jacobian of the transformation z h^ z^^' on lJ{n) is given by 

J(g,z) = |det(a + zc)r2". 

Fix cr, r G C For g G U(n, n) we define the foUowing hnear operator in the 
space C°°(U(n)): 



Pa|r(5)/W = /(^'^0det(a + zc)-"-"det(a + zc) . (1.2) 

The formula includes powers of complex numbers, precise definition is given 
below. In fact, g i— > Pairig) is a well-defined operator- valued function on the 
universal covering group U(n, n)~ of U(n, n). 

The chain rule for Jacobians, 

J{gig2,z)^J{g,,z)J{g2,z^si])^ (I.3) 

implies 

Po-|r(5l)P<T|r(52) = Pa|r(gi.92)- 

In other words, Po-|r is a linear representation of the group \J{n,n)^. 

Observation 1.3 //Recr + Rcr = — n, Im cr = Imr then a representation p^i^. 
is unitary in L^(U(n)). 

This easily follows from the formula for the Jacobian. 

Next, let cr, t be real. We define the Hermitian form on C°°{U{n)) by the 
formula 



(/i, f2)a\r := / / det(l - zu*r{l - z*urfi{z) f2{u) d^{z) d^i{u). (1.4) 

JU(n) JU(n) 

Proposition 1.4 The operators Pairig) preserve the Hermitian form {■^■)^i^. 



Theorem 1.5 For a , t '^'L, the Hermitian form (■, ■)a\T "i-s positive iff integer 
parts of numbers —a — n and t are equal. 

In fact, the domain of positivity is the square — 1 < r < 0, — n < a < —n+ 1 
and its shifts by vectors {—j,j), j € Z , see Figure [5] 

In particular, under this condition, a representation p^i^ is unitary. 

For some values of (<t, r) the form (■,-)^i^ is positive semi-definite. There 
are two the most important cases. 

1. For T — 0, we get highest weight representations (or holomorphic repre- 
sentations). Thus, the Stein-Sahi representations are nearest relatives of holo- 
morphic representations. 

2. For r = 0, cr = 0, —1, —2, . . . , —n we obtain some exotic 'small' 
representations of U(n, n). 

1.3. The structure of the paper. We discuss only group^\J{n,n). 

In Section 2 we consider the case n = 1 and present the Pukanszky classifica- 
tion [37j of unitary representations of the universal covering group of SL(2, R) ci 
SU(1,1). 

In Section 3 we discuss Stein-Sahi representations of arbitrary \J{n,n). In 
Section 4 we explain relations of Stein-Sahi representations and holomorphic 
representations. In Section 5 we give explicit constructions of the Sahi 'unipo- 
tent' representations. 

In Section 6 we discuss some open problems of harmonic analysis. 

1.4. Notation. Let a, u, v E C Denote 

li u — V S Z, then this expression is well defined for all a ^ 0. However, the 
expression is well defined in many other situations, for instance if |1 — a| < 1 
and u, V are arbitrary (and even for |1 — a| = 1, a 7^ 1) 

The norm, \\z\\ of an n x n- matrix z is the usual norm of a linear operator in 
the standard Euclidean space C". 



*A comment for experts. Stein-Sahi representations of a semisimple Lie group G are com- 
plementary series induced from a maximal parabolic subgroup with Abelian nilpotent radical. 

The cases G = U(n,n), Sp(2n,R), G = SO*(4n) (related to tube type Hermitian sym- 
metric spaces) are parallel. The only difficulty is Theorem 13.111 f the expansion of the integral 
kernel in characters, we choose G = U(n,n), because this can be done by elementary tools). 
In the general Hermitian case, one can refer to the version of the Kadell integral |20) from [29) 
(the integrand is a product of a Jack polynomial and a Selberg-type factor. 

For other series of groups, Stein-Sahi representations depend on one parameter, and pic- 
ture is more pure (in particular, inner products for degenerate ('unipotent') representations 
can be written immediately). A BC-analog of Kadell integral is unknown (certainly, it must 
exist, and some special cases were evaluated in the literature, see e.g., [30]). On the other 
hand, Stein-Sahi representations have multiplicity free if -spectra. In such situation, there is 
lot of ways for examination of positivity of inner products, see e.g. |41) . |42) . [S]. 

New elements of this paper are 'blow-up construction' for unipotent representations 
and (apparently) tame models for representations of universal coverings. The representations 
themselves were constructed in works of Sahi. 



We denote the Haar measure on the unitary group U(n) by /x; assume that 
the complete measure of the group is 1. 

The Pochhammer symbol is given by 

,, T{a + n) (a{a + l)...{a + n-l) if n ^ ^^ ^^ 

1 (a) 7 — TY-'-T r if n < 0. 

^ ^ \ (a— l)...(a — n) 

2 Unitary representations of SU(1, 1) 

Denote by SU(1, 1)~ the universal covering group of SU(1, 1). 

In this section, we present constructions of all irreducible unitary represen- 
tations of SU(1, 1)"". According the Bargmann-Pukanszky theorem there are 4 
types of such representations: 

a) unitary principal series; 

b) complementary series; 

c) highest weight and lowest weight representations; 

d) The one-dimensional representation. 

Models of these representations are given below. 

The general Stein-Sahi representations are a strange 'higher copy' of the 
SU(l,l)-picture. 

References. The classification of unitary representations of SL(2,R) ~ 
SU(1, 1) was obtained by V. Bargmann [2j; it was extended to the SU(1, 1)"" by 
L. Pukanszky [37], see also P. Sally [45]. D 

A. Preliminaries 

2.1. Fourier series and distributions. By S^ we denote the unit circle 
\z\ = 1 in the complex plane C We parameterize S^ hy z = e*"^. 

By C°°(5^) we denote the space of smooth functions on S^. Recall, that 

f{ip)^ Y^ a„e"'^ G C°°(S'i) iff |a„| =o(|n|-^) forallL. 

n— — oo 

Recall that a distribution h{(p) on the circle admits an expansion into a 
Fourier series, 



CXD 



h{if) = Y^ 6„e™'^, where |6„| = 0(|n|^) for some L. 

71=0 

For s G M we define the Sobolev space ^{S^) as the space of distributions 

oo 

h{ip) = J2 bne'""^ such that ^ |6„|2(1 + \n\f' < oo. 



n=0 



By definition, W'^{S^) = L'^{S^). For positive integer s ^ k this condition is 
equivalent -^-^h E L'^{S^). Evidently, s < s' implies W D W' . 

1.1. The group SU(1,1). The group SU(1,1) ~ SL(2,R) consists of aU 
complex 2 x 2-matrices having the form 

\ ^), where lap - |5p = 1. 

This group acts on the disc \z\ < 1 and on the circle \z\ — 1 by the Mobius 
transformations 

z ^^ {a + bz)^ (b + az). 

2.3. A model of the universal covering group SU(1,1)~. Recall that 
the fundamental group of SU(1, 1) is Z. A loop generating the fundamental 
group is 

9^(^)-('o^ J^^y fH(27r) = fH(0) = 1. (2.1) 

Some example of multi-valued continuous function on SU(1, 1) are 

aj \b aj 

We can realize SU(1, 1)~ as a subset in SU(1, 1) x C consisting of pairs 

T _ I , cr ) , where e'^ = a. 

Thus, for a given matrix t- _ J the parameter a ranges if the countable set 

a = Ina + 2TTki. 

Define a multiplication in SU(1, 1) x C by 

(5l,cri)° (32,cr2) = (ffl52,CTl +<^2 + c(gi,g2)), 

where c{gi,g2) is the Berezin-Guichardet cocycle, 

c{9i,92) = ln , 

aia2 

Here a^ is the matrix element of 173 — .9152- 



Theorem 1.1 a) 



0.3 _ ^ 

aia2 



< 1, and therefore the logarithm is well defined. 

b) The operation o determines the structure of a group on SU(1, 1) x C 

c) SU(1, 1)~ is a subgroup in the latter group. 



The proof is a simple and nice exercise. 

Now we can define the single- valued function In a on SU(1,1) by setting 
In a := a. 

B. Non-unitary and unitary principal series 

2.4. Principal series of representations of SU(1, 1). Fix p, g G C For 

g € SU(1, 1) define the operator ^^[^(g) in the space C°^{J5^) by the formula 



here we use the notation (|1.5p for complex powers. 

Observation 2.2 a) Tp\q is a well-defined operator-valued function on SU(1, 1)' 
b) It satisfies 

Tp\q{gi)Tp\q{g2) = Tp\q{gig2). 

Proof, a) First, 



{a + hz)-P{a + hz) ^ a'^ ■a-''{l + a-^hz)-'P{l + a-^bz) . 

Since \z\ = 1 and \a\ > \b\, the last two factors are well defined. Next, 

a^P a ^"^ := exp-^ —p \na + qlna> 

and In a is a well-defined function on SU(1, 1)~. 

Proof of b). One can verify this identity for gi, (72 near the unit and refer to 
the analytic continuation. D 

The representations Tp\q{g) are called representations of the principal (non- 
unitary) series. 

Remark, a) A representation T^i^ is a single- valued representation of SU(1, 1) 
iff p — g is integer. 

2.5. The action of the Lie algebra. The Lie algebra 5u(l, 1) of SU(1, 1) 
consists of matrices 

*-^ ^ ) , where a € M, /? £ C. 

p —laj 

It is convenient to take the following basis in the complexification su(l, l)c = 

s[(2,C): 

^. - \ {-: ^ - - (s ;) ■ -^ - (-": s) (») 

These generators act in C°°{S^) by the following operators 

Lo^z— + -{p-q), L^ = — -qz\ L+ = z'—+pz. (2.4) 

az 2 dz dz 



Equivalently, 

Loz" = (n+i(p-g))z", L_z'' = {n - q)z''-\ L+z" ^ (n + p)z"+\ 

(2.5) 



2.6. Subrepresentations. 

Proposition 2.3 A representation Tp\g is irreducible iff p, q <^ Z. 

Proof. Let p, q ^ "L. Consider an Lo-eigenvector z". Then all vectors 
(L+)'^z", {L-Yz" are nonzero. They span the whole space C°°(5^). D 

Observation 2.4 a) If q (E TL, then z'^ , z''^^ , ...span a subrepresentation in 
T I 

b) If p G Z, then z^p, z^^^^ , z^^^"^ , . . . span a subrepresentation in Tp\q. 

Proof of a). Clearly, our subspace is i^-invariant and L+-invariant. On 
the other hand, L~ z'^ = 0, and we can not leave our subspace. D 

All possible positions of subrepresentations of Tp^^ are listed on Figure [T] 

2.7. Shifts of parameters. 

Observation 2.5 If k is integer, then Tp^^q^^ ~ Tp\q. The intertwining op- 
erator is 

Af{z) = zVW- 

A verification is trivial. D 

2.8. Duality. Consider the bilinear map 

given by 

{fij2)^:^rfiie'nf2ie'ndip^^ f^ f,{z)f^{z)-. (2.6) 

27r Jo 27r Jg z 

Observation 2.6 Representations Tp\q and Ti_p\i_q are dual with respect to 
n, i.e., 

n(Tp|,(.g)/i,Ti_p|i_g(g)/2) =n(/i,/2). (2.7) 

Proof. After simple cancelations we get the following expression in the left 
hand side of ([2Jl) 

h('-^)h{'-^)-{a^lz)-^JITlz)-''-l. 
.1 \a + hzj \a + bzj z 



Keeping in mind z = z , we transform 

/ T x-i7 i^^^dz / T \-i/T N-i , fb + az\ /b + az\ 

{a + bz) ^{a + bz) — = (a + 6z) \b + az) Mz = ^ d\ ^ 

z \a + bzj \a + bzj 



quotient-module 



submodule 



a) q is integer, 
submodule 



quotient module 



b) p is integer, 
subm.odule 



subm.odule 



-P 

finite-dim.ensional 
quotient-module 

c) p, q are integer, q + p ^ 1 ; 
quotient-m.odule 



quoticnt-m.odule 



finitc-dim.ensional 
subm.odule 
d) p, q are integer, q + p ^ 1. 

Figure 1: Subrepresentations of the principal series. Black circles enumerate 
vectors z". A representation T^i^ is reducible ifFp G Z or q G Z. 



Now the integral comes to the desired form: 



7^ h{u)f2{u)—. U 



We also define a sesquilinear map 

n* :C°°(S'i) X C°°(S'i) 

by 



27r 



dz 



n*(/i,/2):=n(/i,/2)= / f,{z)h{z)-. (2.8) 

Jo ^ 

Observation 2.7 Representations Tp^q and Ti_q^i_p are dual with respect to 

n*. 

Proof is same. D 

2.9. Intertwining operators. Consider the integral operator 

Iplg :C°°(5'i)^C°°(S'i) 
given by 

^^i«^(") - 2..r(p + ,-f) IJ' - ^^)'''-''-''m T^ (2-9) 

where the function (1 — zu)^p^^^'^^^^ is defined by 

(f - zu){P-i|«-i> := lim (1 - fzu)^P-^l«-i> (2.10) 

The integral converges if Re(p + 5) > — f . 

Theorem 2.8 The map {p\q) >—> Ip\q admits the analytic continuation to a 
holomorphic operator-valued function on C'^ . 

Theorem 2.9 The operator I p\g intertwines Tp\g and Ti_ij\i_p, i.e., 

Tl-p\l-q{g) Ip\q = Ip\q Tp\q{g) ■ 

Corollary 2.10 If p ^ Ia, q ^ Z, then the representations Tp\g and Ti_g\i_p 
are equivalent. 

2.10. Proof of Theorems [HH [2791 

Lemma 2.11 The expansion of the distribution i2. 1 0|) into the Fourier series 
is given by 

= r(p + g-f) Y. vi }rlvi t(-)"- (2.12) 

„~t^ r(p + n)r(g - n) \u) 



Proof. Let Rep, Reg be sufficiently large. Then we write 



and open brackets in ()2.13p . For instance, the coefficient at [z/u]'^ is 

'ML 
k\k\ 



2^ iTIi ^ 2^1(1 -P,i-g;i;i), 



feJsO 

where 2^^! is the Gauss hypergeometric function. We evaluate the sum with the 
Gauss summation formula for 2^1(1), see [18], (2.1.14). D 

Proof of Theorem 12.81 Denote by 

(-1)" 



r(p + n)T{q — n) 

the Fourier coefficients in p.l2p . Evidently, c„ admits holomorphic continuation 
to the whole planeH C^ . 
By m. (1-18.4), 

r(n + a) f, 

— ^ -f ~ nr ° as n -^ ±00. 

Keeping in mind (j2.1ip . we get 

c„ -- const • |n|^~^~'' as n ^ ±00. (2-14) 

Then 

and 

Obviously, this map sends smooth functions to smooth functions. D 

Proof of Corollary 12. 101 In this case, all c„ ^ 0. D 

Proof of Theorem 12.91 The calculation is straightforward, 

Tl-q\l-pig)Ip\qf{u) = 

27r« J\u\=i \ \a + buj J z 

Next, we observe 

-,/ /6 + au\\ , ,n/ f—b + liz 



^The Gamma function r(2) has simple poles at z = 0, —1, —2, . . . and does not have zeros. 
Therefore l/(r(p + n)V(q — n)) has zeros at p = —n, —n — 1, ... and at 5 = n, n — 1, .... 
In particular, if both p, q arc integer and q < p, when / 1 = 0. 
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T 



•» 



Figure 2: The unitary principal series in coordinates 

h = ip-q+l)/2,s = \ip + q-l)/2. 

Equivalently, 

p — h + is, q — 1 — h + is. 

The shift h t-^ h + 1 does not change a representation. Also the symmetry 

s 1-^ —s sends a representation to an equivalent one. Therefore representations 

of the principal series are enumerated by the a semi-strip 0^/i<l, s^O. It 

is more reasonable to think that representations of the unitary principal series 

are enumerated by points of a semi-cylinder {s,h), where s ^ and h is defined 

modulo equivalence h ^ h + k, where h € Z. 



and come to 

i^LO-"(^))'"""'«--)"-"'-'/<='T^ 

Now we change a variable again 

b + aw —b + az 

Z= =^, W= r^ 

a -I- bit; a- bz 

and come to the desired expression 

2« 7|,„|^i \a + bwj w 

2.11. The unitary principal series. 

Observation 2.12 A representation Tp\g is unitary in L'^{S^) iff 

Imp = Im(7, Rep-|-Reg=l. (2-15) 

Proof is straightforward, also this follows from Observation l2.7l D 

C. The complementary series 

2.12. The complementary series. Now let 

0<p<l 0<g<l. (2.16) 
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Consider the Hermitian form on C°°{S ) given by 

{2ttiYT{p + q-l) J J z u 

\z\ = l \u\ = l 

(2.17) 
By (Em, 

(A.a„^J^^-W (2.18) 

Theorem 2.13 // < p < 1, < g < 1, then the inner product {2.17^ is 
positive definite. 

Proof. Indeed, in this case all coefficients 

{P)n (<?)-« 

in (|2.17p are positive. D 

Theorem 2.14 Let 0<p<l, 0<q<l. Then the representation Tp^q is 
unitary with respect to the inner product (•, ■)p\q, i.e., 

{Tp\q{g)fl,Tp\g{g)f2)p\q = {fl:h)p\q- 

Proof. This follows from Theorem 12.91 and Observation l2.7l Indeed, 

{h^h)p\q = n*(/p|q/l,/2) 

and 

^* {Ip\qTp\q{g)fl,Tp\q{g)f2) = ^*{Tl^q\l-p{g)Ip\qfl,Tp\q{g)f2) = 

= n*(/p|,/l,/2) = (/l,/2)p|,. 

Keeping in mind our future purposes, we propose another (homotopic) proof. 

Substitute _ _ 

h + az' b + an' 



a + hz' a + hu' 

to the integral in (|2.17p . Applying the identity 



fb + az'\ fb + au'\ , - ,, i ,_, , ,,7 ^^TT"-^ 

1 - ^-^ —^ = (a + hz') \\ - z'v!)(a + bu') , 
Va + oz'/ \a + bu' J 

we get 

{Tp\q{g)fl,Tplq{g)f2)p\q. □ 

2.13. Sobolev spaces. Denote by "Hp^q the completion of C°°{S^) with 
respect to the inner product of the complementary series. 
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Figure 3: The complementary series. The diagonal is contained in the principal 
series (the segment of the axis Oh on Figl2]). The symmetry with respect to the 
diagonal sends a representation to an equivalent representation. 



First, we observe that the principal series and the complementary series have 
an intersection, see (|2.15p . (|2.16|) . namely the interval 

p + q = 1, <p < 1. 



In this case the inner product (j2.18p is the i^-inner product, i.e., 7ip|i_p ^ 

Next consider arbitrary {p,q), where 0<p<l,0<g<l. By (|2.14p . the 
space Hpiq consists of Fourier series Yl O'nz" such that 



J2 Kl^n'-P-'^ < oo. 



Thus, np,g is the Sobolev space W'(i-p-«)/2(S'1). 

D. Holomorphic and anti-holomorphic representations 

Denote by D the disk \z\ < 1 in C 

2.14. Holomorphic (highest vifeight) representations. Set q = 0, 

\a + bzj 

Since \a\ > \b\, the factor {a + bz)~P is holomorphic in the disk D. Therefore 
the space of holomorphic functions in D is SU(1, 1)"^ -invariant. Denote the 
representation of SU(1, 1)~ in the space of holomorphic functions by Tj'. 

Theorem 2.15 a) For p > the representation T^ is unitary, the invariant 
inner product in the space of holomorphic functions is 



n>0 'J^'" 



(2.19) 



n>0 



n>0 
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b) For p > 1 the invariant inner product admits the following integral repre- 
sentation: 

(/i,/2) -^^ If fi{z)hi^){i-\zfr-'dx{z), 

I" JJ\z\<l 

where dX{z) is the Lebesgue measure in the disk. 

c) For p = 1 the invariant inner product is 

(/i,/2> - ;^ r hie^nhi^d^ =-!./■ /,(^)^(I) ^. (2.20) 

We denote this Hilbert space of holomorphic functions by H+ . 

Proof. The invariance of inner products in b), c) can be easily verified by 
straightforward calculations. 

To prove a), we note that weight vectors z^ must be pairwise orthogonal. 

Next, operators of the Lie algebra su(l, 1) must be skew-self-adjoint. The 
generators of the Lie algebra must satisfy 

(i+)*=L_. 

Therefore, 

(L+z",z"+i) = (z",L_z"+i) 

or 

(7i+p)(z"+\z"+i) = (n+l)(z",2"). 

This implies a). 

If p = 1, then (z", 2") — 1 for n ^ 0, i.e., we get the L^-inner product. D 

The theorem does not provide us an explicit integral formula for inner prod- 
uct in 7i+ if < p < 1. There is another way of description of inner products 
in spaces of holomorphic functions. 

2.15. Reproducing kernels. 

Theorem 2.16 For each p > 0, for any f G Tip , and for each a G D 

{f{z), (1 — za)^^) = f{a) (the reproducing property). (2.21) 

Proof. Indeed, 

(Va„z", VMli^-r') = Va„%u"(z",2") = Va„y" = f{u). D 

In fact, the identity (|2.2ip is an all-sufRcient definition of the inner product. 
We will not discuss this (see [lU', f3T), and prefer another way. 

2.16. Realizations of holomorphic representations in quotient spaces. 

Consider the representation T_i|_i_p of the principal series, 

& + az\ - ■^- =— --1-P 



T-i|-i-,/W = / f^^) (a + 6z)-i(a + 6z) 
\a^ bz J 
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The corresponding invariant Hermitian form in C°°{S^), is 

{fij2)-i\-i-,^j^ f I il^zu)-PMz)h{u)--. (2.22) 

[^-^iV J\z\ = lJ\u\ = l Z U 

(we write another pre-integral factor in comparison with (|2.17p l. The integral 
diverges for p > 1. However, we can define the inner product by 

f (£k if n > 

^ ' [0 ifn<0 

the latter definition is valid for all p > 0. 

We denote by L C C°°[S^) the subspace consisting of series X^tko*^"-^"- 
This subspace is SU(1, l)-invariant and our form is nondegenerate and positive 
definite on the quotient space C°°{S^)/L. 

Next, we consider the intertwining operator 
as above (but we change a normalization of the integral), 

U\^i-pfiu) = -^ / (1 - zur^fiz) - 

27ri J|2|=i z 

The kernel of the operator is L and the image consists of holomorphic functions. 
Observation 2.17 a) The operator /-i|-i-p is a unitary operator 

c°°{s^)/L^n+. 

h) The representation r_i|_i_p in C°°{S^)/L is equivalent to the highest 
weight representation Tf 

2.17. Lowest v^reight representations. Now set p = 0, q > 0. Then 
operators Tqi^ preserve the subspace consisting of 'antiholomorphic' functions 
J2n<o^-n,z"'. Denote by T~ the corresponding representation in the space of 
antiholomorphic functions. These representations are unitary. 

We omit further discussion because these representations are twins of highest 
weight representations. 

E. The bloviT-up trick 

Here we discuss a trick that produces 'unipotent' representations of U(n, n) 
for n > 2, see Subsection [ 



2.18. The exotic case p = 1, q = 0. In this case, 

Ti|o = T+®Tf. 
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submodule 



submodule 



b) 




Figure 4: a) The strueture of the representation r^io. 

b) Ways to {p,q) = (1,0) from different directions give origins to different in- 
variant Hermitian forms on Ti\q. By our normahzation, the inner product is 
positive definite in the gray triangle and negative definite in the white triangle. 
Therefore coming to (1,0) from the grey triangle we get a positive form. 



Let us discuss the behavior of the inner product of the complementary series 
near the point {p\q) = (1|0), 



'Jl,f2)p\q 



(27ri)2 



dz 



\z\ = l 



(1-Z^)^P-1|«-1}/1(Z)/2(Z)- 



(2.23) 



Consider the limit of this expression asp ^ 1, g ^ 0. The Fourier coefficients 
of the kernel are the following mcromorphic functions 



Cn{P-,(l) = 



i-irrip + q-i) 



T{q — n)T(p + n) 

Note that 

1. Cn{p, q) has a pole at the line p + q = 1; 

2. for n ^ 0, the function c„(p, q) has a zero on the line q = 0; 

3. for n < 0, the function c„(p, q) has a zero at the line p = 0. 

Thus our point {p,q) = (1, 0) lies on the intersection of a pole and of a zero 
of the function c„ (p, q) . Let us substitute 

p = 1 + es q ~ et, where s + t ^ 

to Cn{p, q) and pass to the limit as £ ^ 0. Recall that 

(-1)" 



r(z) 



n! (z + n) 



o(i). 



as z 



~n, where n = 0, 1, 2, 



(2.24) 
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Therefore we get 

i-n- ifn^O 

hm c„(l + es,et) = < "^^^ 

In particular, for s = we get r^ -inner product, and for t = we get 
T]^ -inner product. Generally, 

oo oo oo —1 

E^o ^ — ' ^ — ' ' t + s ^ — ' t + s ^ — ' 

n— — oo n— — oo n— n— — oo 

Therefore we get a one-parametric family of invariant inner products for Tiiq- 
However, all of them are linear combinations of two basis inner products men- 
tioned above {t — and s — 0). 

3 Stein— Sahi representations 

Here we extend constructions of the previous section to the groups G :— U(n, n). 
The analogy of the circle S^ is the space U(n) of unitary matrices. 

A. Construction of representations 

3.1. Distributions £cr\T- Let z be an n x n matrix with norm < 1. For 
(T G C, we define the function dct(l — z)'^ by 






det(l - zy := det 
Extend this function to matrices z satisfying ||z|| $; 1, dct(l — z) 7^ by 



dct(l - zy := lim dct(l - u)" . 

u — *2:, ||u|| <1 

The expression det(l — z)" is continuous in the domain ||z|| ^ 1 except the 
surface det(l — z) = 0. 

Denote by det(l — z)^"^'^^ the function 

det(l - z)^'"l^> :== det(l - zf dct(l - z)^. 

We define the function i.„\^^(yg) on the unitary group U(n) by 

4|^(z) := 2-('^+^)" det(l - z){'^l^>. (3.1) 

Obviously, 

i^\^{h-^zh)^l„\^{z) for z, /i e U(n). (3.2) 
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Lemma 3.1 Let e''^^ , ..., e"''", where ^ f/'fc < 27r, he the eigenvalues of 
z G V{n). Then 



t,\Az) - exp{^(a - r) ^(V-fe - tt)} J] sin-+^ ^. (3.3) 



Proof. It suffices to verify the statement for diagonal matrices; equivalently 
we must check the identity 



(l_e^^)H-}=exp{^(a-T)(V'-7r)}. 



sm — . 
2 



We have 

i(l-e^'/') = exp{^(^-^)}sin|. 

Further, both the sides of the equaUty 

2-^(1 - e'^)- = exp{^a(^ - ^)} sin'^ |, 

are real-analytic on (0, 27r) and the substitution ip = tt gives 1 in both the sides. 

D 

3.2. Positivity. Let Re(cr + r) < 1. Consider the sesquilinear form on 
C°°(U(n)) given by 



{flj2)a\r= e,\r{zu-')fl{z)f2{u)dtl{z)df,{u). (3.4) 

J Jv{n)xV{n) 

For cr, T G M this form is Hcrmitian, i.e., 



(/2,/i)ct|t — (/l, /2)CT|r 

Observation 3.2 For fixed /i, /2 G C°°(U(n)), f/izs expression admits a mero- 
morphic continuation in a, t to the whole C^. 

This follows from general facts about distributions; however, this fact is a 
corollary of the expansion of the distributions i^i^. in characters, see Theorem 
13.111 This expansion implies also the following theorem: 

Theorem 3.3 For cr, r G K \ Z, the inner product ^3.4^ is positive definite (up 
to a sign) iff integer parts of —a — n and t are equal. 

The domain of positivity is the union of the dotted squares on Figure [S] 
For a, T satisfying this theorem, denote by Hair the completion of C°° (U(n)) 
with respect to our inner product. 

3.3. The group U(n, n). Consider the linear space C" C" equipped with 
the indefinite Hcrmitian form 

{v ®w,v' ® w'} ^ {v, wOc-eo - {w, w')o0C" , (3.5) 
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Shift 




1. The dotted squares correspond to unitary representations pcr\T- 

2. Vertical and horizontal rays in the south-west of Figure correspond to nondegen- 
erate highest weight and lowest weight representations. Fat points correspond to 
degenerated highest and lowest weight representations, and also to the unipotent 
representations. The point (cr, r) = (0,0) corresponds to the trivial one-dimensional 
representation. 

3. In points of the thick segments, we have some exotic unitary sub-quotients. 

4. The shift (a, r) ^^ (ct + 1, r — 1) send a representation p^\^ of SU(n, n)~ to an 
equivalent representation. 

5. The permutation of the axes (t, cr) i-^ (c, r) gives a complex conjugate repre- 
sentation. 

6. The symmetry with respect to the point (— n/2, — n/2) (black circle) gives a 
dual representation (for odd n this point is a center of a dotted square; for even n 
this point is a common vertex of two dotted squares). 

7. For (T + T = n (the diagonal line) our Hermitian form is the standard L^-product. 

8. Linear (non-projective) representations of U(n, n) correspond to the family of 
parallel lines cr — r £ Z. 

Figure 5: Unitarizability conditions for U(n,n). The case n = 5. 
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where (•, •) is the standard inner product in C". Denote by U(n,n) the group 
of Unear operators in C" © C" preserving the form {•,•}. We write elements of 

a b^ 



this group as block {n + n) x {n + n) matrices g '■— [ ,] ■ By definition, such 

matrices satisfy the condition 

Lemma 3.4 The following formula 

z^ z^s^ ■.= {a + zc)-\b + zd), zeU(n), .g=r ^]e\J{n,n) (3.7) 

determines an action of the group U(n,n) on the space U(n). 

Proof is given in Subsection 13.201 

3.4. Representations po.|r of U(n,n). Denote by U(ri, n)"" the universal 
covering of the group U(n,n), see for details Subsection l3.18l Fix a, t £ C We 
define an action of U(n, n)"" in the space C°°(U(n)) by the linear operators 

P.\rig)f{z) = /(zl^l) det^-"-^l-"-->(a + zc). (3.8) 

We must explain the meaning of the complex power in this formula. First, 

a + zc = (1 + zca'' )a 

The defining equation p.6p implies ||ca^^|| < 1. Hence, for all matrices z 
satisfying ||2:|| ^ 1, complex powers of 1 + zca~^ are well defined. Next, 

det(a)"""^'"""'' := exp|-(n + r) Indct a - {n + aJhidctaX 

It is a well-defined function on lJ{n, n)"". We set 

det(a + zc)-"-^|-"-'" :=det[(l + zca-i)-"-^|-"-'"j det(a)-"-^l-"-'" 

3.5. The Stein Sahi representations. 

Proposition 3.5 The operators p^^^lg) preserve the form (■, ■)a\T- 
Proof is given in Subsection 13.231 

Corollary 3.6 For a, t satisfying the positivity conditions of Theorem \3.3\ the 
representation p^\^ is unitary in the Hilbert space Ti^w- 

3.6. The degenerate principal series. 

Proposition 3.7 Let Re(p + cr) = — n, Inicr = Inir. Then the representation 
p^^^ is unitary in L'^(\J{n)). 
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m„ 7713 m2 m-i 

nnnBnnnnnnnnBBnnnBD 

Figure 6: A ''Maya diagram' for signatures. We draw the integer 'line' and fill 
the boxes rrii, . . . , 7ti„ with black. 



Proposition 3.8 

(/ij/2)o-|t 



n;=ir(a+r+j) 



a^ — 7i — r 



const- / fi{u)f2iu)dfi{u) 

JV(n) 



3.7. Shifts of parameters. 
Proposition 3.9 For integer k, 

Pa+k\T-k - (defy)'' -p^i^ 
The intertwining operator is the multiplication by the determinant 

F{z) ^ F{z)det{z)''. 
This operator also defines an isometry of the corresponding Hermitian forms. 

B. Expansions of distributions £cr|r in characters. Posi- 
tivity 

3.8. Characters of U(n). See Weyl's book gg. The set of finite di- 
mensional representations of U(n) is parameterized by collections of integers 
(signatures) 

m : TOi > ?7i2 > • ■ • > m„. 

The character Xm of the representatiorO TTm (a Schur function) corresponding 
to a signature m is given by 

detfc,-i2 „|e™''^'=| 
detfej=i^2,...,n|eHJ ^>1"'] 

where e^'^*' are the eigenvalues of z. Recall that the denominator admits the 
decomposition 

det{e*(j-i)'^'=} =]J (e*'^' -e''^'=). (3.10) 

The dimension of tt™ is 

dmiTTm = Xm(l) = ^^TTi H ■ (3.11) 



nn 



^Explicit constructions of representations of U(n) are not used below. 
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3.9. Central functions. A function F{z) on lJ{n) is called central if 

F{h-^zh)=F{z) for all z, /i e U(n). 

In particular characters and i^iT- are central functions. 

For central functions F on U(n), the following Weyl integration formula 
holds 

n-)'^M^)-(^ / •■■ / ^(diag(e*,...,e''^"))x 

U(n) 0<?/Ji<27r O<-0„<27r 



X 



J|(e.^™_e.V'.)| -Qrf^^^ (3.12) 



ni<k fc— 1 



where diag(-) is a diagonal matrix with given entries. 

Any central function F e L'^{\J{n)) admits an expansion in characters, 

^ — ^m 

where the summation is given over all signatures m and the coefhcients Cm are 
L^-inner products 



Cm = / F{z)Xm{z)dfl{z). 

J\}(n) 

Note that Xm — Xm* , where 

m* := (n — 1 — m„, . . . , n — 1 — TO2, n — 1 — mi) 

Applying formula (|3.12p . explicit expression (|3.9p for characters, and formula 
(|3.10P for the denominator, we obtain 



^ ' ' Fldiag^e*'^! e'^-Mx 



(2.)".! y ■•• J 4diag{e^^S--^e'^"}) 

O<-0i<27r 0<-!/'7i<27r 



det |e*0-i)'/''=| det |e^*'"^'^'=| TT d^Jfe. (3.13) 



X 

■■' ' ■ ...... ^^^ 



Let F[z) be multiplicative with respect to eigenvalues, 

k 

(for, instance F = ^cr|T> see ([33])). Then we can apply the following simple 
lemma (see e.g. [28]). 
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Lemma 3.10 Let X be a set, 

n n 

TT /(xfc) det {ui{xk)} det {vi{xk)}W dxj = 

■^■^ kl—l....n k,l—l,...n -^ -^ 

fc=l ' ■ j=l 

= n! det < / f{x)ui{x)vm{x) dx> (3.14) 

/,m— l,...,n L / V J 



3.10. Lobachevsky beta-integrals. We wish to apply Lemma [3. 101 to 
functions ia\T- For this purpose, we need for the following integral, see [T5] . 
3.631,1,3.631,8, 

, sin^-^Me'^^d^= ^,' iwolrr ;,^n/9V (3-1^) 

r((^ + fc + i)/2)r((^-fe + i)/2) 

It is equivalent to the identity (|2.12|) . 

In a certain sense, the integral (13.211) is a multivariate analog of the Lobachevsky 
integral. On the other hand, ()3.2ip is a special case of the modified Kadell in- 
tegral [ini. 

3.11. Expansion of the function i„^^ in characters. 

Theorem 3.11 Let Re(cr + t) < 1. Then 

4|r(3) = 

(-l)"("-i)/2sin"(7ra)2-('^+^)" A . 

(3.16) 

n 

= (_l)"(«-i)/22-('-+^)« Jl r(ff + r + j) X 

(_l)E"b n (m„-m/3) . 

X E ^^ ^^^^^^^^ XM . (3.17) 

m I Jl r(cr — nij + n)r(r + rrij + 1) J 

Proof is contained in Subsection l3.13l For the calculation we need for Lemma 
13. 131 proved in the next subsection. 

3.12. A determinant identity. Recall that the Cauchy determinant (see 
e.g. [22]) is given by 

ki \xk + yi j n i^k + yi) 

23 



The following version of the Cauchy determinant is also well known. 
Lemma 3.12 



/ 



1 



det 



1 



\ 



xi+bi 
1 

Xl+b2 



\ 1 

\xi+6„_i 



X2+bl 

1 

2:2+62 



X3+bl 

1 

X3+b2 



Xn + bl 

1 
Xn + b2 



X2+bn 



1 1 / 



6/3) 



n {^k + ^a) 



(3.19) 



Proof. Let A be the Cauchy determinant p.lSp . Then 



/ XX 



yi 



yi 



2/1 A 



+yi 



xi+y2 



\xi + 



+yn 



x2+yi 

X2+y2 



X2+yn 



yi 



i+yi \ 



Xn+y2 



X-n+Vn 



We take lim yiA and substitute Va+i = ba- 

The following determinant is a rephrasing of [22", Lemma 3. 

Lemma 3.13 



n 



/ 



det 



1 



1 



1 



x,+b, 

xi+ai 

{xi+ai){xx+a2) 

(xi+bi)(xi+b2) 


X2+ai 

X2+bl 

(x2+ai){x2+a2) 

(x2+bi){x2+b2) 


X3+ai 

X3+bl 

{x3+ax)(x3+a2) 

{x3+bi){x3+b2) 




x-a+ai 

x„+bi 
(2;„+ai)(a:„+a2) 
(x„+6i)(3;„+b2) 




n (:ri+a„) 

ls:m<r— 1 


n {x2+a„,) 

n {x2+br„) 


n (x3+a^) 


- 


n (^n+a„ 

i: l<Tn<Ti-l 


.) 


n (Xl+brn) 


n {X3+brr^) ■ 


n {Xr^+br, 
1: lsJmsJre-1 







n {^i^-^i) n 


-1 


10 nn\ 





n i^k+bfi) 

Proof. Decomposing a matrix element into a sum of partial fractions, we 
obtain 



{xk + ai)...{xk +a„) 



{Xk 



{xk + ha) 



= 1 



E 



Y{j<Mj-bi3) 



1 



n 



l^,3^a iij^QJ#^ 



{hj - hp) Xk + bfi 



24 



Therefore the {a + l)-th row is a hnear combination of the following rows: 



1 1 

1 1 



xi+bi X2+bi 



x„+bi n 



( -^- -^ 

\ Xi+ba: X2 + ba 

Thus our determinant equals 



Xn+ba 






( \ \ 



1 \ 



x\-\-h\ x-2^h\ 



\x^+l 



Xn+b\ 



1 / 

Xn+bc' 



l+6a X-i^-ba 

and we refer to Lemma [3. 121 

3.13. Proof of Theorem 13.111 We must evaluate the inner product 



n 



U(7^ 



C|r(5)Xm(5)rfAi(5)- 



Applying ()3.13p . we get 



(2^y 



0<-i/'fc<27r 



i=i 



det {e-*™'='^'}. det {e'^^-^'^^'^^din. (3.21) 



i=\ 



By Lemma r3.10[ we reduce this integral to 

1 



(27r)" l^fcj^n 



det /(fc,j), 



where 



27r 



/(fc, j) = e"*('""^)''/2 / sin'"-^(V'/2) • exp{i( {a + t)/2 + fc - 1 - TOj)} d^- 
Jo 

We apply the Lobachevsky integral ()3.15|) and get 

_ 2i-'^-^7rr(o- + T + 1) (-l)fc-i-'". 
*- '^'^ r(o- + /s-mj)r(T-fc + mj+2) 

Applying standard formulas for F-function, we come to 

/(fc, j) = 2i— -r(f7 + T + 1) sin(-a^) . ^Jt^+ "'^' - ^ + 1) ^ 



2i"'""^r(cr+r+l) sin(-cr7r) 



r(— (7 + TOj — n + 1) 
r(T + TOj - n + 2) 



-fc + 2) 




{ — U + TOj 


-n+ l)„_fc 


(r + TOj - 


- n + 2)„_fc 
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The factors outside the box do not depend on on k. Thus, we must evaluate 

the determinant 

{-a + nij -n+ l)n-fc 

i^k,j^n (t + rrij — n + 2)„_j. 

Up to a permutation of rows, it is a determinant of the form described in Lemma 
[37131 with 

Xj — rrij , Gj = —a — n + j, b = T — n + j + 1. 

After a rearrangement of the factors, we obtain the required result. D 

3.14. Characters of compact groups. Preliminaries. First, recall 
some standard facts on characters of compact groups, for details, see e.g. [21], 
9.2, 11.1. 

Let K he a, compact Lie group equipped with the Haar measure /x, let 
Ijl{K) = 1. Let TTi, 7r2, ...be the complete collection of pairwise distinct ir- 
reducible representations of K. Let xii X2, . . . be their characters. Recall the 
orthogonality relations. 



{Xk,Xi)L^K)= Xk{h)xi{h) dfj,{h) ^ Sk,i (3.22) 



IK 

and 



where * denotes the convolution on the group, 

u*v{g)= / u{gh^ )v{h)dfi{h). 
Jk 

Consider the action of the group K x K in L'^{K) by the left and right shifts 

(fci,A;2): f{9)^f{K'gk2). 

The representation oi K x K in L'^{K) is a multiplicity free direct sum of 
irreducible representations having the form vr^ ® tt^ , where tt^ denotes the dual 
representation, 

L\K)^^TTk(E)K. (3.24) 

k 

Denote by Vk C L"^ (K) the space of representation tt^ ® tt^ . Each distribution 
/ on ivT is a sum of 'elementary harmonics'. 



/ = Efe/'' A-el4.. 



The projector to a subspace Vk is the convolution with the corresponding 
character, 

f^:T7^—f*Xk (3.25) 

(in particular, /'^ is smooth). 
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Observation 3.14 Let f be a function on U(n), / = J2m^"if"^, where f" S 
a)feC^{Vin)) iff 



ll/'"IU^=o(E™') for all L. 

b) f is a distribution on U(n) iff there exists L such that 



,2' 



Proof: Note that / G L^(U(n)) iff J2 Wf^Wh < oo. Denote by A be the 
second order invariant Laplace operator on U(n). Then A/™ ~ q{m)f"^, where 
g(m) = Yl rn^j + ■ • ■ is an exphcit quadratic expression in m. For / G C°° we 
have A^/ € C°°\ this imphes the first statement. Since q(m) has a finite number 
of zeros (one) , the second statement follows from a) and the duality. D 

3.15. Hermitian forms defined by kernels. Let S be a central distri- 
bution on K satisfying ^(g^^) = ^(g). Consider the following Hermitian form 
onC°°{K) 



(/i,/2)-// ^{gh-^)f^{h)h{9)d^x{h)d^i{g). (3.26) 

Consider the expansion of S in characters 

k 

Lemma 3.15 

(/i, /2) - E IT^ I fiWlmMh). (3.27) 

^ dun TTfe Ju(«) 

Proof. The Hermitian form (|3.26p is K x X-invariant. Therefore the 
subspaces Vk ~ tt/c (g) tt^ must be pairwise orthogonal. Since iTk (Ei tt^ is an 
irreducible representation oi K x K, it admits a unique up to a factor K x K- 
invariant Hermitian form. Therefore it is sufficient to find these factors. 

Set fi = f2 — Xk- We evaluate 



// yicfcXfc(g/i ^) ] Xk{h)xkig) dn{g) dn{h) = 

JJkxk \ ,. / 



Cfe 



KxK y-^ J dimTTfc 

using (|3:22l) and ((3:231) . D 

3.16. Positivity. Let Re(cr + r) < 1. Consider the sesquilinear form on 
C°°(U(n)) given by 



{flj2)a\r^ i,\rizu-')fliz)f2{u)dt,iz)dtliu), (3.28) 

J JU(n)xU(n) 

where the distribution ^^It is the same as above. 
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Observation 3.16 For fixed fi, /2 S C°°{l]{n)), the expression {fi,f2)a\T o-d- 
mits a meromorphic continuation in a, t to the whole C^. 

Proof. Expanding /i, /2 in elementary harmonics 

m m 

we get (see Lemma [3.15p 

-^ dimTTmJuH 

where the meromorphic expressions for Cm were obtained in Theorem l3.11l The 
coefhcients Cm have polynomial growth in m. On the other hand, ||/'"|| rapidly 
decrease, see Observation 13.141 Therefore, the series converges. D 

Proof of positivity. Corollary 13. 61 We look at the expression p.l6p . 
It suffices to examine the factor 

because signs of all the remaining factors are independent on nij. Let n ^ "L 
and a G (0,1). Then 

■ J.I ^ . /+1> ifn^>0, 

signl (n + a) = < , 
^ ^ ' \(-l)", ifn<0 

Therefore p.29p is positive whenever integer parts of t and —a — n equal. D 

3.17. The L^-limit. Proof of Proposition [HH Thus, let cr + r = ~n. 

Then 

f fj r((T + T + j) j t„\r = const 'E (dim 7rm)Xm 

Indeed, in this case F-factors in (|3.16p cancel, and we use p. lip . 
Keeping in mind (|3.27p . we get Proposition 13.81 

C. Other proofs 

Here we prove that the operators p^lr preserve the inner product determined 
by the distribution ia\r. 

3.18. The universal covering of the group U(n, n). The fundamental 
group of U(n, n) iCi 

7ri(U(n,n)) ~ Z ® Z. 



^By a general theorem, a real reductive Lie group G admits a deformation retraction to its 
maximal compact subgroup K. In our case, K = U(n) X U(n) and 7ri(U(n)) = Z. 
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The universal covering U(n, n)~ of U(n, n) can be identified with the the set 
il of triples 

, I , s, < i- e \]{n,n) X C X 
satisfying the conditions 

det(a)=e^ det(d) = e*. 

The multiplication of triples is given by the formula 

{9i,si,ti)o {g2,S2,t2) = {9192, Si +S2 +c+{gi,g2),ti + ^2 +c^{9i,92)), 
where the Berezin cocycle c is given by 

c^{9i,92) = trln(a^^a3a^^), c~{gi,g2) = tTlnid^^dsd^^); 

here g^ — gig2, and gj —I •' / I . It can be shown that \\ai 0302 ^ 111 < 1, 

\^j "■jj 
therefore the logarithm is well defined. On the other hand, 

g.3 ^ g«i+s2+c+ (31,32) ^ det(ai)det(a2)det(aj:^a3a^i) = det(a3) 

This shows that the il is closed with respect to multiplication. 

For details, see [3Tj . 

In particular, det(a) is a well-defined single-valued function on U(n, n)"^ . In 
our notation, it is given by 

{9,s,t) H-> s. 

3.19. Another model of \J{n,n). We can realize \J{n,n) as the group 
of {n + n) X [n + n)-matrices g — ( <■ ] satisfying the condition 

<-. o)"*^(-. 0) <"»' 

3.20. Action of U(n,n) on the space U(n). Proof of Lemma 13.41 . 

We must show that for 

z G U(n) and 9 = ( j I ^ U(n, n) 

we have 

zM ;= (a + zc)-\b + zd) e U(n, n) (3.31) 

For z e U(n), consider its graph graph{z) C C" ©C". It is an n-dimensional 
linear subspace, consisting of all vectors v ® vz, where a vector-row v ranges 
in C". Since z G U(n), the subspace graph{z) is isotropiqfl with respect to the 



*A subspace y in a linear space is isotropic with respect to an Hermitian form Q if Q 
equals on V. 
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Hermitian form „ _. j . Conversely, any n-dimensional isotropic subspace in 

C" C" is a graph of a unitary operator z G U(7i). 

Thus we get a one-to-one correspondence between the group U(n) and the 
Grassmannian of n-dimensional isotropic subspaces in C" © C". 

The group U(n,n) acts on the Grassmannian and therefore U(ri, n) acts on 

the space U(n). Then (|3.3ip is the exphcit expression for the latter action. 

Indeed 

a b 



(v(Bvz] 



, , — v{a + zc) © v{b + zd) 



We denote ^ := v{a + zc) and get 

£. © ^{a + zc)~'^{b + zd) 



and this completes the proof of Lemma 13.41 D 

Thus, U(n) is a V{n, n)-homogeneous space. We describe without proof (it 
is a simple exercise) the stabilizer of a point z = 1. It is a maximal parabolic 
subgroup. 

In the model (|3.30p it can be realized as the subgroup of matrices having 

the structure 

'a 

It is a semidirect product of GL(n, C) and the Abelian group M" . 

In our basic model the stabilizer of z = 1 is the semi-direct product of two 
subgroups 

1 /„. , „*-! „. „*-l\ 

where g e GL(n,C), (3.32) 

where r = T*. (3.33) 



2 \a~ a* ^ a + a* ^ 



and 



1 + iT iT 
-iT 1 - iT 



3.21. The Jacobian. 
Lemma 3.17 For the Haar measure fi(z) on U(n), we have 

H{z^s]) ^ I det-2"(a + zc)\ ■ ^i{z). (3.34) 

Proof. A verification of this formula is straightforward, we only outline the 
main steps. First, J(g, z) := \ det^'^" (a + zc)\ satisfies the chain rule (|1.3p . Next, 

the formula (I3.34p is valid for g G \J{n,n) having the form I , J, where u, 

V G U(7i). Indeed, the corresponding transformation oi u i-^ u"'"' is u i-^ a^^ud, 
its Jacobian is 1. 

Therefore we can set z = 1, z^^l — 1. Now we must evaluate the determinants 
of the differentials of maps z i— > z^^l at z = 1 for g given by (|3.32p and (|3.33[) . 
In the second case the differential is the identity map, in the first case the 
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differential is dz i-^ a*{dz)a. We represent a as pAq, wliere A is diagonal with 
real eigenvalues and p, q are unitary. Now the statement becomes obvious. D 
3.22. The degenerate principal series. Proof of Proposition 13.71 

Thus, let Re(CT + t) = -n, fm(o-) = Im(T) = s. Then 

det(a + MC)-"-''!-"-^ = | det(a + uc)|-"^2iSg»(r-a)Argdet(a+«c)^ 

where Arg(-) is the argument of a complex number. Therefore 

{Ta\Ti9)fl,T„\T{g)f2)L-^{Vin)) = 

2 



/i(mI»1)/2(wM) det(a + uc)-"-'^l-"-^ dn{u) = 

U(«) 



/i (ulsl )/2 (mM ) I det(a + wc) r^" d^(u) 
u(«) 

and we change the variable z — u^^' keeping in mind Lemma 13.171 D 

3.23. The invariance of the kernel. Proof of Proposition! 



Lemma 3.18 The distribution I^-^t- satisfies the identity 

4|^(mM(w[91)*) = £„ir{uv*) det(a + uc)^-^^-"'' det(a + vc)^-''^-^K (3.35) 

Proof. This follows from the identity 

1 - wM(uM)* = {a + ucy^{l -uv*){a + vc)*^'^, where g e U(n,n), 
which can be easily verified by a straightforward calculation (see e.g. [3T]). D 

Proof of Proposition I3.5I First, let Re(cr+r) < 1. Substitute hi = u^f, 
/i2 = ^2 to the integral 



(/l,/2).|r= // 4|r(/ll/^2)/l(/il)/2(/l2)dA*(/ll)ci/x(/l2). 

J JU(n)xU(n) 

By the lemma, we obtain 

4|^(uiM2) det(a + uic)^-'l-''>| dct(a + M2c)r''l"^^ x 

U(n)xU(n) 

X /i(ui)^^(M^|det(a + uic)|~^"|det(a + U2c)p^"d^(ui)d^(u2) = 

= {Pa\T{9)h,Pa\T{g)h)a\T- 

Thus, our operators preserve the form (•, •)cr|r- 

For general cr, t e C, we consider the analytic continuation. D 

3.24. Shift of parameters. Proof of Proposition [379l First, we recall 
Cartan decomposition. For ti ^ . . . ^ t„ denote 

/cosh(ii) ...\ /sinh(ti) 

CH(t) := 

V 



cosh(t2) ■ • ■ . SH(t) := '^ sinh(t2 



The following statement is well known 
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n-1 a 

^ • • • • • ► 

Figure 7: Conditions of positivity of liolomorphic representations (,„ (the 
^ Berezin-Wallach sef). 

Proposition 3.19 Each element g G \J{n,n) can be represented in the form 
ui 0\ fCRit) SIi{t)\ (u2 

V2 



^=( JISHW CH(i))io ^-1 (^-2^) 



for some (uniquely determined) t and some ui, U2, Wi, W2 G U(n). 

Now we must show that the operator f{z) i-^ det(z)/(z) intertwines p^^^ 
and Pa+iW-i A straightforward calculation reduces this to the identity 

det(a + zc) _ det(zM) 
det (a + zc) A-et{z) 

which becomes obvious after the substitution ()3.36|) . 
Also, 

and this easily implies the second statement of Proposition 



4 Hilbert spaces of holomorphic functions 

Theorem 13.31 exhaust the cases when the form (•, •)o.|r is positive definite on 
C°°(U(7i)). However there are cases of positive semi-definiteness. They are 
discussed in the next two sections. 

Set r = 0. In this case, our construction produces holomorphic represen- 
tationqj of U(n, n). Holomorphic representations were discovered by Harish- 
Chandra (holomorphic discrete series, [17j ) and Berezin (analytic continuations 
of holomorphic discrete series, [3]). They are discussed in numerous texts (for 
partial expositions and further references, see e.g. [10], [31]), our aim is to show 
a link with our considerations. 

4.1. The case r = 0. Substituting r = 0, we get the action 

P<T|o(.g) f{z) = /(z'sl) det(a + zc)-"det(a + zc) """". 
The Hermitian form is 



(/i,/2).|o=/ / dei{l -z*uYf^{z)f2{u)dp{z)dp{u). 

Jvin) Jv{n) 



^or highest weight representations, 
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Theorem 4.1 The form (/i,/2)ct|o ** positive semi- definite iff cr is contained 
in the set 

a = 0, —1, . . . , —{n — 1), or a < — (n — 1). 



This means that all coefficients Cm in the formula p.27p are non-negative, 
but some coefficients vanish. In fact the proof (see below) is the examination of 
these coefficients. 

Under the conditions of the theorem we get a structure of a pre-Hilbert space 
in C°°(U(n)). Denote by Jia the corresponding Hilbert space. 

Next, consider the action of the subgroup U(n) x U(n) in H.^. We must get 
an orthogonal direct sum 

meOc 



Some of summands of (I3.24p disappear, when we pass to the quotient space; 
actually the summation is taken over a proper subset f2o- of the set of all repre- 
sentations. The next theorem is the description of of the set ^„. 

Theorem 4.2 a) If a < — (n - 1), then 

Do- := jm : m„ ^ OJ-. 

b) If o — —n + a, where a = 1, 2, . . . , n — 1, n, then 

r^cr = |m : m„ = 0, m„_i = 1, . . . , nin-a+i = a-l\. 

Proofs^ Substitute r = to (|3T7| . 






(-1)2: "b n {mc.-mp) 

\ — . Xm(.9) 



m L ]^ r(cr — TUj + n) T{mj + 1) 
j=i ' 



(4.1) 



(-l)"("-i)/2sin"(7i-a)2-('^)" 



^^{ n ("^"-"^/3)n 



r(-cr + 'mj-n + l) 



T(m, + 1) 



(4.2) 



Xm(5) (4.3) 



^"This is the original Berezin's proof, he started from explicit expansions of reproducing 
kernels 1 14.611 . 
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n n n n 



1 

nnnnBnnnnnBnn 



all'boxes-are'empty 

b) 

obligatory-part 



n n n n 



12 3 4 



nnBDnnnann 



all-boxes-are-empty 

Figure 8: 'Maya diagrams' for signatures of harmonics in liolomorphic repre- 
sentations. 

a) A general case, a < n — I. 

b) Degenerate case. Here cr = — (n — l) + 5. 

We have T{mj + 1) = oo for rrij < 0. Therefore the corresponding fractions 
in (|4.3p are zero, and the expansion of £^10 has the form 



Yl CmXm. (4.4) 



m: rrin ^0 

Let US list possible cases. 

Case 1. If c < —n — 1, then all coefficients Cm are positive, see ()4.3p : in the 
line (|4.2p poles of the Gamma-functions cancel with zeros of sines. 

Case 2. If c ^ —n — 1 is non- integer, then all the coefficients Cm are 
non-zero, but they have different signs. 

Case 3. Let a be integer, a ^ —n + 1. Consider a small perturbation of a, 

a = —n + a + e. 

In this case we get an uncertainty in the expression (|4.ip : 



n"=i r(a - TTij + e) 



0. 



The order of the pole of the numerator is n — a. However order of a pole in 
the denominator ranges between n — a and n according to m. If the last order 
> n — a, then the ratio is zero. The only possibility to get order of a pole 
= ri — a is to set 

m„ = 0, m„_i = 1, . . . , nin-a+i = 0. (4.5) 



Thus the coefficients Cm are nonzero only for signatures satisfying (J4.5P : they 
are positive. 



34 



We omit a discussion of positive integer a (the invariant inner product is not 
positive). D 

4.2. Intertwining operators. Denote by B„ the space of complex n x n- 
matrices with norm < 1. 

Consider the integral operator 

Iaf{z) = f det(l - zh*Yf{h) dtx{h), z e B„. 

JU{n) 

It intertwines Po-lo with the representation jO_„i „_g., denote the last represen- 
tation by ^(j 

e.(.9)/W = /(^'^l)det(a + zc)^ 

The /(j-image 7i° of the space Ha consists of functions holomorphic in B„. The 
structure of a Hilbert space in the space of holomorphic functions is determined 
by the reproducing kernel 

Kc,iz, u) ^det{l- zu*y. (4.6) 

4.3. Concluding remarks (without proofs). 

a) For (T < — (2n — 1), the inner product in H° can be written as an integral 



(/i,/2) = const / /i(z)/2(z) det(l - zz*)—'''dzdz. 

b) For a < n — 1 the space 7i° contains all polynomials. 

c) Let (7 = 0, —1, . . . , — {n — 1). Consider the matrix 

/ d d \ 

dzii ' ' ' dzin 

A= : ■.. : 

^dZnl ' ' ' dZn 

The space 'H°_,_-^s consists of functions / satisfying the partial differential 
equation 

(detA)/(z)-0. 

The space H", where ct = 0, —1, . . . , —{n — 1), consists of functions that are 
annihilated by all (— cr+l) x (— a-|-l)-minorsof the matrix A. Also, ?i° contains 
all polynomial satisfying this system of equations. 
In particular, the space Hq is one-dimensional. 

5 Unipotent representations. 

Here we propose models for 'unipotent representations' of Sahi [43] and Dvorsky- 
Sahi, 0-0. 
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obligatory-part 
1 a-1 



D D D ■ D D D 



D D ■ D D ■ D 



Figure 9: Maya diagram for signatures e Zj] here j is the number of black boxes 
at the left of the 'obhgatory part'. 



a) 



m2. 



m2. 



\ o o 

c) 



( • ) o — e — e- 



TOi 



"12. 



h) 



d) 



-^ 



o o o o 
o o * o o o o 

m2. 



TOi 



TTii 



Figure 10: The case n = 2. 

a) a = 0. The decomposition of i^(U(2) into a direct sum. 

b) a = 1. White circles correspond to the big subrepresentation Wtaii- The 
quotient is a direct sum of two subrepresentations. 

c) a = 2. The quotient is one-dimensional. 

d)0<r<l,cr = — n. The invariant filtration. The subquotients are unitary. 
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5.1. Quotients of p^^^ at integer points. Set 

r = 0, (T ~ —n + a, where a = 0, 1, . . . , n — 1. (5-1) 

For j = 0, 1, . . . , n — a denote by Zj the set of all signatures m of the form 

m = (toi,.. .,m„_Q_j, a- l,a- 2,. . . ,0, m„_j+i, . ..m„) 

Denote by Vm the U(n) x U(ri,)-subrepresentation in C°°(U(n)) correspond- 
ing a signature m, see Subsection l3.14l 

Theorem 5.1 The subspace 

Wtau:= V™ C C°°(U(n)), 

is U(n, n) -invariant. 

b) The quotient C°° (\J (n)) /Wtaii is a sum n ^ a + 1 subrepresentations 

The representation of U(n, n) in each Wj is unitary. 

We formulate the result for a = separately. In this case Wtaii ~ 0. 

Theorem 5.2 The representation p-n\o is a direct sum ofn+1 unitary repre- 
sentations Wj, where ^ j ^ n. We have Vm C Wj if the number of negative 
labels ruk is j . 

In particular, we get a canonical decomposition of L^(\J(n)) into a direct 
sum of {n + I) subspaces. 

Proof is given in the next subsection. 

5.2. The blow-up construction. [^ The distribution i^^^. depends mero- 
morphically in two complex variables a, r. Its poles and zeros are located at 
cr G Z and in r £ Z. For this reason, values of ^o-|t at points {a, r) G Z^ generally 
are not uniquely defined. Passing to such points from different directions, we 
get different limita^4 

Thus, set 

a = —n + a + se, t = te where (s, i) 7^ (0, 0). (5.2) 



^^The case U(l, 1) was considered above in Subsection 12.181 

^^A remark for an expert in algebraic geometry: we consider blow up of the plane C^ at 
the point (— n + «, 0). 



37 



Substituting this to (|3.17p . we get 

n 
i^r.+c.+es\et = (-l)"("-l)/22-(-+-)" J] r(-71 + ae{s + t) + k) X 

k=l 

(_l)E"b Yl {ma -nib) . 

xE\-n '-^^^^ xM\. (5.3) 

Theorem 5.3 a) Let s ^ —t. Then there exist a limit in the sense of distribu- 
tions: 

r*(z):=lim£_„+„+„|,,(z), (5.4) 

£— »0 

In other words, the function (cr|T) f— > ^o-|t- has a removable singularity at e = 
on the line 

a = —n + a + ss, t = et, where e G C 

h) Denote by c^s : t) the Fourier coefficients of t'^'^ . If m is in the 'tail', 
i.e., m ^ UZj, then Cm(s : t) ~ 0. 

c) Moreover, i?*'* admits a decomposition 

r' = y f^- £,, (5.5) 

3=0 ^ > 

where £j is of the form 

^2 = 51 amXm, (5.6) 

where Om f^o noi depend on s, t. 

d) For each j all coefficients a^ in I15.6\) are either positive or negative. 

Proof. For the numerator of (|5.3p we have the asymptotic 

n 

Y[ T{-n + ae{s + t) + k) = Ce-"+"(s + t)-"+" + 0(£-"+"+i), e ^ 0. 

Next, examine factors of the denominator, 

( Ai{mk){et)-^ ifmfe<0 
r{a + es — mk)T{st + mk + l)^<A2{mk) ii ^ ruk < a , e^O 

[A3{mk){esy^ ii nik ^ a, 

where Ai, A2, A^ do not depend on s, t. Therefore, the order of the pole of 
denominator Jlfc of (|5.3p is 

number of mj outside the segment [0,a — 1] 
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The minimal possible order of a pole of the denominator is ti — a. In this case, 
Cm has a finite nonzero limit, of the form 

number of mp^ ^ a _ j-nuinbcr of m^ < 0. 

Cm(s : t) = A(m) ■ 

If an order of pole in the denominator is > n — a, then Cmis : t) — 0. This 
corresponds to the tail. 

We omit to watch the positivity of c„i{s : t). 

Formally, it is necessary to watch the growth of Cjn{s : t) as m — > cx) and 

the growth of 

d , 

■T^Cmi-n + a + es, et) 
ae 

to be sure that (|5.4p is a limit in the sense of distributions. This is a more-or-less 
trivial exercise on the Gamma-function. D 

There are many ways to express £■' in the terms of £^'*. One of variants is 
given in the following obvious proposition. 

Proposition 5.4 The distribution £,j is given by the formula 

1 B^ 

£.W = ^7^(1 + 0"""^'^*W „ (5.7) 



j! dti 



t=o 



5.3. The family of invariant Hermitian forms. Thus, for (ct, r) = 
(—71 + a,0) we obtained the following families of p_„_|_Q|o-invariant Hermitian 
forms 

R'-\hJ2) ■■= 1 1 r-\zu*)fi{z)7^dfi{z)dfi{u) (5.8) 

J Jv{n)xV{n) 

and 

Qj(/i,/2):= // £,izu*)Mz)'h(^d^liz)d^liu) (5.9) 

■/ JU(n)xU(n) 

They are related as 
A form £j is zero on 

Yj := Wtail © {®^^JW^) 

and determines an inner product on Wj ~ C°°{\J(n))/Yj. 
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6 Some problems of harmonic analysis 

6.1. Tensor products pc\r ® Pa'\T' • Nowadays the problem of decomposi- 
tion of a tensor product of two arbitrary unitary representations does not seem 
interesting. We propose several informal arguments for reasonableness of the 
problem in our case. 

a) For n = 1 it is precisely the well-known problem of decomposition of 
tensor products of unitary representations of SL(2,R)'" ~ SU(1, 1)", see [55] . 
[23], [39], [ig, [32]. 

b) Decomposition of tensor products pafi ® Pa' fi of holomorphic representa- 
tions is a well-known combinatorial problem, see [T9j . 

c) Tensor products Po-,o <8>Po|t' are Berezin representations, see [4], [47], [27] . 

d) All problems a)-c) have interesting links with theory of special functions. 

e) There is a canonical isomorphism V\ 

P-n/2\-n/2®P-n/2\^n/2 =i i^ (U(n, n)/GL(n, C)) . (6.1) 

Thus, we again come to a classical problem, i.e., the problem of deconrposition 
of L^ on a pseudo-Riemannian symmetric space G/H, see [TT], [35 1^3 Gen- 
eral tensor products Pa.|T ® P<j'\t' can be regarded as deformations of the space 
L2(U(n,n)/GL(n,C)). 

6.2. Restriction problems. 

1. Consider the group G* :— U(n, n) and its subgroup G :— 0{n,n). The 
group G has an open dense orbit on the space U(n), namely 

G/H:^0{n,n)/0{n,C)- 

The restriction of the representation p^n/2\-n/2 to G is equivalent to the rep- 
resentation of G in L'^{G/H). Restrictions of other p^^^ can be regarded as 
deformations of L'^{G/H). 

The same argument produces deformations of L^ on some other pseudo- 
Riemannian symmetric spaces. Precisely, we have the following variants. 

2. G* = U(2n, 2n), G/H = Sp(n, n)/Sp(2n, C). 

3. G* = V{n,n), G/H = S0*(2n)/0(n, C). 

4. G* = U(2n, 2n), G/H = Sp(4n, M)/Sp(2n, C). 

5. G* = U(p + q,p + q), G/H = \J{p,q) x \J{p,q)/\J{p,q). In this case, 
G/H^V{p,q). 

6. G* — U(n, n), G = GL(ri, C). In this case, we have {n + 1) open orbits 
G/Hp = GL(n, C)/U(p, n - p). 



^''indeed, U(n) ~ U(n, n)/P, where P is a maximal parabolic subgroup in U(n, n). The 
group U(n,7i) has an open orbit on U(n,n)/P X U(n,n)/P, the stabilizer of a point is ~ 
GL(n,C). 

^■^In a certain sense, the Plancherel formula for L'^{G/H) was obtained in [l], [6]. However 
no Plancherel measure, nor spectra are known. The corresponding problems remain open. 
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7. G* — U(n, n) X U(n, n), G = U(n, n). This is the problem about tensor 
products discussed above. 

6.3. The Gelfand— Gindikin programm. Recall the statement of the 
problem, see |13) . [34j . Let G/ H be a pseudo-Riemannian symmetric space. The 
natural representation of G in L'^{G/H) has several pieces of spectrum. There- 
fore, L'^{G/H) admits a natural orthogonal decomposition into direct summands 
having uniform spectra. The problem is: to describe explicitly the corresponding 
suhspaces or corresponding projectors. 

In Subsection 15.11 we have obtained a natural decomposition of L^(U(n)) 
into (n + 1) direct summands. Therefore in the cases listed in Subsection \6.2\ 
we have a natural orthogonal decompositions of L^(G/H). 

In any case, for the one-sheet hyperboloid U(l,l)/C* we get the desired 
construction (see Molchanov [SJ, pS]). 

6.4. Matrix Sobolev spaces? Our inner product {■,-)^^^ seems to be 
similar to Sobolev-type inner products discussed in Subsectio ri2.13l However 
it is not a Sobolev inner product, because the kernel det(l — zu*)^'^^'^^ has a 
non-diagonal singularity. 

Denote 



Let F be a distribution on U(n), let F = ^ F^ be its expansion in a series 
of elementary harmonics. We have 

' ^-^ dimTTm 

n 

^ E{ll^-lli^n(i + K-l)'}<^' (6-2) 



i=i 



where ||Fj„||l2 denotes 

WFrr 



/U(«) 



.ih)f d^i{h)) 



1/2 



Our Hermitian form defines a norm only in the case \s\ < 1, but (|6.2[) makes 
sense for arbitrary real s. Thus we can define a Sobolev space Hg on U(n) of 
arbitrary order. 

Author does not know applications of this remark, but it seems that it can 
be useful in the following situation. 

First, a reasonable harmonic analysis related to semisimple Lie groups is the 
analysis of unitary representations. But near 1980 Molchanov observed that 
many identities with special function admit interpretations on " physical level of 
rigor" as formulas of non-unitary harmonic analysis. Up to now, there are no 
reasonable interpretations of this phenomenon (but formulas exist, see, e.g. [7], 
see also |27j. Section 1.32 and formula (2.6)~(2.15) ). In particular, we do not 
know reasonable functional spaces that can be scene of action of such analysis. 
It seems that our spaces H5 can be possible candidates. 
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